The Kolmogorov Sampler

Given noisy observations Xi = θi + Zi, i = 1, . . . , n, with noise Zi iid ∼ N(0, σ), we wish to recover the signal θ with small mean-squared error. We consider the Minimum Kolmogorov Complexity Estimator (MKCE), defined roughly as the n-vector θ̂(X) solving the problem min Y K(Y ) subject to ‖X − Y ‖2l2n ≤ σ 2 · n, where K(Y ) denotes the length of the shortest computer program that can compute the finite-precision n-vector Y . In words, this is the simplest object that fits the data to within the lack-of-fit between θ and X that would be expected on statistical grounds. Suppose that the θi are successive samples from a stationary ergodic process obeying Eθ 1 < ∞. Under a regularity condition on the process, we show that θ̂ behaves, for large n, as a single sample from the posterior L(θ|X). In a sense, then, θ̂ is a universal empirical Bayesian procedure. Parallel results hold in other settings. Suppose we have binary data Xi which arise as Xi = θi ⊕ Zi, where each unknown θi is binary-valued, and ⊕ denotes the exclusive-or of Bernoulli noise and θ. Define MKCE in this setting as the Kolmogorov-simplest binary string matching the observed data except in n places. Suppose that θ follows a stationary ergodic process. Under a regularity condition on the process, we show that the MKCE behaves as a sample from the posterior distribution. Our proof combines ideas from information theory to get what seems an essentially novel conclusion: that when the loss level is equal to the true underlying noise level, optimal lossy compression of random process data must exhibit reconstructions which look like samples from the correct Bayesian posterior. Our proof relies on a key idea from ergodic theory, namely the concept of B-process; this furnishes the ‘regularity condition’ on stationary ergodic processes mentioned earlier. We also rely on d̄ and ρ̄ distances. Structure theorems about the class of B-processes allow us to conclude that ρ̄-close to every B-process are other B-processes where an optimal codebook for Shannon lossy compression can be computed by a short program. Since codebook performance is ρ̄-continuous, the Kolmogorov-simplest representation matching the noisy data to within the noise level must compress at least as well as optimal Shannon compression for such processes.